{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2020 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/GoogleCloudPlatform/keras-idiomatic-programmer/blob/master/books/deep-learning-design-patterns/Workshops/Junior/Deep%20Learning%20Design%20Patterns%20-%20Workshop%20-%20Chapter%201.ipynb\">\n",
    "<img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Learning Design Patterns - Code Labs\n",
    "\n",
    "## Lab Exercise #5 - Get Familiar with Model Design Patterns\n",
    "\n",
    "## Prerequistes:\n",
    "\n",
    "    1. Familiar with Python\n",
    "    2. Completed Chapter I: Models by Design\n",
    "\n",
    "## Objectives:\n",
    "\n",
    "    1. Create a model design template\n",
    "    2. Construct the stem and classifier component\n",
    "    3. Construct the learner component\n",
    "    4. Construct blocks with max pooling\n",
    "    5. Construct blocks with feature pooling\n",
    "    6. Compare maxpooling to feature pooling training differences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Design Template\n",
    "\n",
    "Let's create a model template based of the macro-architecture, which includes:\n",
    "\n",
    "    1. stem\n",
    "    2. learner\n",
    "    3. classifier\n",
    "    \n",
    "<img src='https://github.com/GoogleCloudPlatform/keras-idiomatic-programmer/blob/master/books/deep-learning-design-patterns/Workshops/Junior/macro.png?raw=true'>\n",
    "    \n",
    "You fill in the blanks (replace the ??), make sure it passes the Python interpreter.\n",
    "\n",
    "You will need to:\n",
    "\n",
    "    1. Add the activation function to the stem.\n",
    "    2. Pass the group parameters for each group to the group method.\n",
    "    3. Add global average pooling to the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import Input, Model\n",
    "from tensorflow.keras.layers import Dense, Conv2D, ReLU, BatchNormalization, GlobalAveragePooling2D, MaxPooling2D\n",
    "\n",
    "def stem(inputs):\n",
    "    outputs = Conv2D(32, (3, 3), strides=(1, 1), padding='same')(inputs)\n",
    "    outputs = BatchNormalization()(outputs)\n",
    "    # Add activation function\n",
    "    # HINT: implement Conv-BN-RE (post-activation batch normalization)\n",
    "    outputs = ?? \n",
    "    return outputs\n",
    "\n",
    "def learner(inputs, groups):\n",
    "    outputs = inputs\n",
    "    for group_params in groups:\n",
    "        # Pass the group parameters as python kwargs\n",
    "        # HINT: remember the string ** parameter syntax?\n",
    "        outputs = group(outputs, ??)\n",
    "    return outputs\n",
    "\n",
    "def group(inputs, **blocks):\n",
    "    outputs = inputs\n",
    "    for block in blocks:\n",
    "        pass\n",
    "    return outputs\n",
    "\n",
    "def classifier(inputs, n_classes):\n",
    "    # Flatten and reduce the feature maps to single pixel each.\n",
    "    # Hint: it is method with 'Global' in the name.\n",
    "    outputs = ??\n",
    "    outputs = Dense(n_classes, activation='softmax')(outputs)\n",
    "    return outputs\n",
    "\n",
    "# Create the input tensor\n",
    "inputs = Input((32, 32, 3))\n",
    "\n",
    "# Assemble the components of the model\n",
    "outputs = stem(inputs)\n",
    "outputs = learner(outputs, [ {'blocks':  [{'n_filters': 64}]}, \n",
    "                             {'blocks':  [{'n_filters': 128}, {'n_filters':128}]}\n",
    "                           ])\n",
    "outputs = classifier(outputs, 10)\n",
    "\n",
    "# Put the model together\n",
    "model = Model(inputs, outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the model architecture using summary method\n",
    "\n",
    "It should look like below:\n",
    "\n",
    "```\n",
    "Model: \"model\"\n",
    "_________________________________________________________________\n",
    "Layer (type)                 Output Shape              Param #   \n",
    "=================================================================\n",
    "input_8 (InputLayer)         [(None, 32, 32, 3)]       0         \n",
    "_________________________________________________________________\n",
    "conv2d_6 (Conv2D)            (None, 32, 32, 32)        896       \n",
    "_________________________________________________________________\n",
    "batch_normalization_6 (Batch (None, 32, 32, 32)        128       \n",
    "_________________________________________________________________\n",
    "re_lu_5 (ReLU)               (None, 32, 32, 32)        0         \n",
    "_________________________________________________________________\n",
    "global_average_pooling2d (Gl (None, 32)                0         \n",
    "_________________________________________________________________\n",
    "dense (Dense)                (None, 10)                330       \n",
    "=================================================================\n",
    "Total params: 1,354\n",
    "Trainable params: 1,290\n",
    "Non-trainable params: 64\n",
    "_________________________________________________________________\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Out the Convolutional Blocks\n",
    "\n",
    "Next, we will complete the learner component by:\n",
    "\n",
    "    1. Design the group method\n",
    "    2. Design the block method using max pooling\n",
    "    \n",
    "You will need to:\n",
    "\n",
    "    1. Extract the blocks parameters for the group.\n",
    "    2. Extract the number of filters parameter for the block.\n",
    "    3. Add max pooling in block to downsample size of feature maps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def group(inputs, **blocks):\n",
    "    outputs = inputs\n",
    "    # Extract the blocks parameters from kwargs blocks\n",
    "    # Hint: the parameter blocks is a dictionary, and 'blocks' is the key\n",
    "    blocks  = ??\n",
    "    for block_params in blocks:\n",
    "        outputs = block(outputs, **block_params)\n",
    "    return outputs\n",
    "\n",
    "def block(inputs, **block):\n",
    "    # Extract the number of filters from the kwargs block\n",
    "    # HINT: key is n_filters\n",
    "    n_filters = ??\n",
    "    outputs = Conv2D(n_filters, (3, 3), strides=(1, 1), padding='same')(inputs)\n",
    "    outputs = BatchNormalization()(outputs)\n",
    "    outputs = ReLU()(outputs)\n",
    "    \n",
    "    # Add max pooling layer to reduce feature maps by 75%\n",
    "    # HINT: defaults to strides=(2, 2), but you can specify it anyways\n",
    "    outputs = MaxPooling2D((2, 2))(outputs)\n",
    "    return outputs\n",
    "\n",
    "inputs = Input((32, 32, 3))\n",
    "outputs = stem(inputs)\n",
    "outputs = learner(outputs, [ {'blocks':  [{'n_filters': 64}]}, \n",
    "                             {'blocks':  [{'n_filters': 128}, {'n_filters':128}]}\n",
    "                           ])\n",
    "outputs = classifier(outputs, 10)\n",
    "model_a = Model(inputs, outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the model architecture using summary method\n",
    "\n",
    "It should look like below:\n",
    "\n",
    "```\n",
    "Layer (type)                 Output Shape              Param #   \n",
    "=================================================================\n",
    "input_2 (InputLayer)         [(None, 32, 32, 3)]       0         \n",
    "_________________________________________________________________\n",
    "conv2d_1 (Conv2D)            (None, 32, 32, 32)        896       \n",
    "_________________________________________________________________\n",
    "batch_normalization_1 (Batch (None, 32, 32, 32)        128       \n",
    "_________________________________________________________________\n",
    "re_lu_1 (ReLU)               (None, 32, 32, 32)        0         \n",
    "_________________________________________________________________\n",
    "conv2d_2 (Conv2D)            (None, 32, 32, 64)        18496     \n",
    "_________________________________________________________________\n",
    "batch_normalization_2 (Batch (None, 32, 32, 64)        256       \n",
    "_________________________________________________________________\n",
    "re_lu_2 (ReLU)               (None, 32, 32, 64)        0         \n",
    "_________________________________________________________________\n",
    "max_pooling2d (MaxPooling2D) (None, 16, 16, 64)        0         \n",
    "_________________________________________________________________\n",
    "conv2d_3 (Conv2D)            (None, 16, 16, 128)       73856     \n",
    "_________________________________________________________________\n",
    "batch_normalization_3 (Batch (None, 16, 16, 128)       512       \n",
    "_________________________________________________________________\n",
    "re_lu_3 (ReLU)               (None, 16, 16, 128)       0         \n",
    "_________________________________________________________________\n",
    "max_pooling2d_1 (MaxPooling2 (None, 8, 8, 128)         0         \n",
    "_________________________________________________________________\n",
    "conv2d_4 (Conv2D)            (None, 8, 8, 128)         147584    \n",
    "_________________________________________________________________\n",
    "batch_normalization_4 (Batch (None, 8, 8, 128)         512       \n",
    "_________________________________________________________________\n",
    "re_lu_4 (ReLU)               (None, 8, 8, 128)         0         \n",
    "_________________________________________________________________\n",
    "max_pooling2d_2 (MaxPooling2 (None, 4, 4, 128)         0         \n",
    "_________________________________________________________________\n",
    "global_average_pooling2d_1 ( (None, 128)               0         \n",
    "_________________________________________________________________\n",
    "dense_1 (Dense)              (None, 10)                1290      \n",
    "=================================================================\n",
    "Total params: 243,530\n",
    "Trainable params: 242,826\n",
    "Non-trainable params: 704\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_a.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Redesign the Convolutional Block\n",
    "\n",
    "Next, we will redesign the convolutional block by:\n",
    "\n",
    "    1. Replacing max pooling with feature pooling.\n",
    "           \n",
    "You will need to:\n",
    "\n",
    "    1. Add a feature pooling convolution\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def block(inputs, **block):\n",
    "    n_filters = block['n_filters']\n",
    "    outputs = Conv2D(n_filters, (3, 3), strides=(1, 1), padding='same')(inputs)\n",
    "    outputs = BatchNormalization()(outputs)\n",
    "    outputs = ReLU()(outputs)\n",
    "    \n",
    "    # Add a feature pooling convolution to reduce feature map size by 75%\n",
    "    # HINT: It is in the strides and padding\n",
    "    outputs = Conv2D(n_filters, (3, 3), ??)(outputs)\n",
    "    outputs = BatchNormalization()(outputs)\n",
    "    outputs = ReLU()(outputs)\n",
    "    return outputs\n",
    "\n",
    "inputs = Input((32, 32, 3))\n",
    "outputs = stem(inputs)\n",
    "outputs = learner(outputs, [ {'blocks':  [{'n_filters': 64}]}, \n",
    "                             {'blocks':  [{'n_filters': 128}, {'n_filters':128}]}\n",
    "                           ])\n",
    "outputs = classifier(outputs, 10)\n",
    "model_b = Model(inputs, outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the model architecture using summary method\n",
    "\n",
    "It should look like below:\n",
    "\n",
    "```\n",
    "Layer (type)                 Output Shape              Param #   \n",
    "=================================================================\n",
    "input_25 (InputLayer)        [(None, 32, 32, 3)]       0         \n",
    "_________________________________________________________________\n",
    "conv2d_29 (Conv2D)           (None, 32, 32, 32)        896       \n",
    "_________________________________________________________________\n",
    "batch_normalization_29 (Batc (None, 32, 32, 32)        128       \n",
    "_________________________________________________________________\n",
    "re_lu_28 (ReLU)              (None, 32, 32, 32)        0         \n",
    "_________________________________________________________________\n",
    "conv2d_30 (Conv2D)           (None, 32, 32, 64)        18496     \n",
    "_________________________________________________________________\n",
    "batch_normalization_30 (Batc (None, 32, 32, 64)        256       \n",
    "_________________________________________________________________\n",
    "re_lu_29 (ReLU)              (None, 32, 32, 64)        0         \n",
    "_________________________________________________________________\n",
    "conv2d_31 (Conv2D)           (None, 16, 16, 64)        36928     \n",
    "_________________________________________________________________\n",
    "batch_normalization_31 (Batc (None, 16, 16, 64)        256       \n",
    "_________________________________________________________________\n",
    "re_lu_30 (ReLU)              (None, 16, 16, 64)        0         \n",
    "_________________________________________________________________\n",
    "conv2d_32 (Conv2D)           (None, 16, 16, 128)       73856     \n",
    "_________________________________________________________________\n",
    "batch_normalization_32 (Batc (None, 16, 16, 128)       512       \n",
    "_________________________________________________________________\n",
    "re_lu_31 (ReLU)              (None, 16, 16, 128)       0         \n",
    "_________________________________________________________________\n",
    "conv2d_33 (Conv2D)           (None, 8, 8, 128)         147584    \n",
    "_________________________________________________________________\n",
    "batch_normalization_33 (Batc (None, 8, 8, 128)         512       \n",
    "_________________________________________________________________\n",
    "re_lu_32 (ReLU)              (None, 8, 8, 128)         0         \n",
    "_________________________________________________________________\n",
    "conv2d_34 (Conv2D)           (None, 8, 8, 128)         147584    \n",
    "_________________________________________________________________\n",
    "batch_normalization_34 (Batc (None, 8, 8, 128)         512       \n",
    "_________________________________________________________________\n",
    "re_lu_33 (ReLU)              (None, 8, 8, 128)         0         \n",
    "_________________________________________________________________\n",
    "conv2d_35 (Conv2D)           (None, 4, 4, 128)         147584    \n",
    "_________________________________________________________________\n",
    "batch_normalization_35 (Batc (None, 4, 4, 128)         512       \n",
    "_________________________________________________________________\n",
    "re_lu_34 (ReLU)              (None, 4, 4, 128)         0         \n",
    "_________________________________________________________________\n",
    "global_average_pooling2d_8 ( (None, 128)               0         \n",
    "_________________________________________________________________\n",
    "dense_8 (Dense)              (None, 10)                1290      \n",
    "=================================================================\n",
    "Total params: 576,906\n",
    "Trainable params: 575,562\n",
    "Non-trainable params: 1,344\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_b.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparsion\n",
    "\n",
    "Finally, lets do a comparison of the models between max pooling and feature pooling by training a small number of epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "\n",
    "Let's get the tf.Keras builtin dataset for CIFAR-10. These are 32x32 color images (3 channels) of 10 classes (airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks). We will preprocess the image data (not covered yet)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import cifar10\n",
    "import numpy as np\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "x_train = (x_train / 255.0).astype(np.float32)\n",
    "x_test  = (x_test / 255.0).astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results - Max Pooling\n",
    "\n",
    "Let's train the model which uses max pooling for 3 epochs.\n",
    "\n",
    "Because it just a few epochs, you test accuracy may vary from run to run. For me, it was 57.2%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_a.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "model_a.fit(x_train, y_train, epochs=3, batch_size=32, validation_split=0.1, verbose=1)\n",
    "model_a.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results - Feature Pooling\n",
    "\n",
    "Let's train the model which uses feature pooling for 3 epochs.\n",
    "\n",
    "For me, my result wasn 65.95%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_b.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "model_b.fit(x_train, y_train, epochs=3, batch_size=32, validation_split=0.1, verbose=1)\n",
    "model_b.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observation\n",
    "\n",
    "Since we replaced the max pooling with feature pooling, you see a modest increase in the training time to additionally train the feature pooling layer.\n",
    "\n",
    "If you compare on an epoch by epoch between the two models, you generally will see little difference on the first two epochs, and then gradually the feature pooling will increase in accuracy faster than the max pooling. The delay is due to the fact that it takes a bit of training to teach the feature pooling layer to optimize pooling the feature maps.\n",
    "\n",
    "In otherwords, it is random and thus starts worse than max pooling (static algorithm) and gradually learns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of Lab Exercise"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
